-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support BITSHUFFLE option of Blosc >= v0.7.3 #899
Conversation
Blosc 0.7.3 now treats the blosc "shuffle" option as an Integer rather than a Bool. This allows the use of all three of blosc's shuffling options: no shuffling (0), byte shuffling (1), or bit shuffling (2). The `shuffle` field of the BloscFilter struct is not restricted to Bool, but it was effectively being converted to Bool via `!=(0)` in the `blosc_filter()` function of the H5Zblosc plugin before passing to the `blosc_compress()` function. This prevented the use of bit shuffling because `2 != 0` is `true` which is the same as 1 (byte shuffling). The H5Zblosc plugin no longer converts the given value to Bool and also validates the value against the three supported shuffle option values. These values are also imported into the H5Zblosc module so users can refer to them as one of: - H5Zblosc.NOSHUFFLE - H5Zblosc.SHUFFLE - H5Zblosc.BITSHUFFLE
Thank you! |
We really should consider a strategy of shifting everything to some version of Blosc2 possibly starting with incorporation of Blosc2.jl into JuliaIO. |
Agreed |
https://github.com/waralex/Blosc2.jl says it's WIP, I wonder what's still missing. Is he open potentially to moving to JuliaIO? |
With regard to this filter we should also review how similar it is to these reference implementations: |
These are all great comments vis a vis Blosc2, but perhaps they would be better placed in a new Blosc2 feature request issue so they don't get lost here. Just saying... |
The operating issue is JuliaIO/Blosc2.jl#6 |
Blosc 0.7.3 now treats the blosc "shuffle" option as an Integer rather
than a Bool. This allows the use of all three of blosc's shuffling
options: no shuffling (0), byte shuffling (1), or bit shuffling (2).
The
shuffle
field of the BloscFilter struct is not restricted to Bool,but it was effectively being converted to Bool via
!=(0)
in theblosc_filter()
function of the H5Zblosc plugin before passing to theblosc_compress()
function. This prevented the use of bit shufflingbecause
2 != 0
istrue
which is the same as 1 (byte shuffling).The H5Zblosc plugin no longer converts the given value to Bool and also
validates the value against the three supported shuffle option values.
These values are also imported into the H5Zblosc module so users can
refer to them as one of:
closes #285